Marketing Research

MKTG 440

Questionnaire Design

Thoughts on SB LX?

Housekeeping

I will give feedback on GA1 before Wednesday
GA2: Plan to finalize logistics around implementing the focus group this week (recruiting, identifiying a time/place, etc.)
Questions?

Calendar

Mon

Tue

Wed

Thu

Fri

Sat

Sun

2/9Survey
Design

2/10

2/11Survey
Design

2/12

2/13

2/14

2/15

2/16Sampling

2/17

2/18Sampling

2/19

2/20Thinking
Forward

2/21

2/22GA2 due

2/23Exam
Review

2/24

2/25Exam 1

2/26

2/27

2/28

3/1GA3 due

3/2Technology
Day

3/3

3/4No Class

3/5

3/6

3/7

3/8

3/9No Class

3/10

3/11No Class

3/12

3/13

3/14

3/15

Will discuss TF more on Wednesday
Exam prep materials next week

Agenda

Common problems with survey questions
Measurement considerations
Getting the best answers
Survey structure

😀

Formulation biases (wording sensitivity)

Ambiguity

Which brand of shampoo do you use?

Resolve the ambiguity

Which brand of shampoo do you use?

More specific:

Which brand or brands of shampoo have you personally used at home during the last month? If you have used more than one brand, please list all the brands that apply.

Specify who, what, when, and where.

Which brand of shampoo do you use?

The Ws	Defining the question
Who	The respondent • It is not clear whether the question relates to the individual respondent or the respondent’s total household.
What	The brand of shampoo • It is unclear how the respondent is to answer this question if more than one brand is used.
When	Unclear • The time frame is not specified. The respondent could interpret it as the shampoo used this morning, this week, or over the past year.
Where	Not specified • At home, at the gym, on the road?

Leading questions

Retail experts believe that all consumers should comparison shop. Do you agree?

○
Agree

○
Disagree

A leading question introduces bias by suggesting desired answers.

To what extent do you agree or disagree that all consumers should comparison shop?

○
Strongly
Disagree

○
Somewhat
Disagree

○
Neutral

○
Somewhat
Agree

○
Strongly
Agree

Loaded questions

Should Americans buy imported automobiles that take away American jobs?

○
Yes

○
No

A loaded question suggests a socially desirable answer or involves an emotionally charged issue.

Please select the number that best expresses your tendency when shopping for a car:

Buy Domestic
Cars

○
1

○
2

○
3

○
4

○
5

○
6

○
7

Buy Foreign
Cars

Leading vs. loaded questions

Leading questions encourage or guide the respondent towards a desired answer.

5 stars across the board means we are doing a good job. How would you rate your service today?

○
1

○
2

○
3

○
4

○
5

Loaded questions assume something about the respondent or the consequences of the response.

Have you stopped speeding on the highway?

○
Yes

○
No

Double-negatives

Do you oppose not allowing the passage of Proposition 418?

○
Yes

○
No

Two “wrongs” don’t make a “right” in surveys.

Do you support or oppose Proposition 418?

○
Support

○
Oppose

Measurement issues

Nominal issues

Not collectively exhaustive

Which kind of computer do you own?

○
Windows PC

○
Apple

A nominal scale should have a category for every possible response.

Which kind of computer do you own? Check all that apply.

☐
I do not own
a computer

☐
Windows PC

☐
Apple Mac

☐
Other

Double-barreled questions

Do you think Sprite is a tasty and refreshing soft drink?

○
Yes

○
No

A double-barreled question asks about two things at once, making responses ambiguous.

Do you think Sprite is a tasty soft drink?

○
Yes

○
No

Do you think Sprite is a refreshing soft drink?

○
Yes

○
No

Ordinal issues

Ambiguous responses

In a typical month, how often do you shop in department stores?

○
Never

○
Occasionally

○
Sometimes

○
Often

○
Regularly

Better:

In a typical month, how often do you shop in department stores?

○
Less than
once

○
1 or 2
times

○
3 or 4
times

○
More than
4 times

Ambiguous responses

How many classes are you taking this semester?

○
4

○
5

○
6

○
Other

Better:

How many classes are you taking this semester?

○
3 or fewer

○
4

○
5

○
6

○
7 or more

Interval/ratio issues

Bad intervals

I am excited for Spring Break.

○
Strongly
Disagree

○
Disagree

○
Neutral

○
Strongly
Agree

Use pre-configured Likert Scales (remember that equal distance is an assumption):

I am excited for Spring Break.

○
Strongly
Disagree

○
Somewhat
Disagree

○
Neutral

○
Somewhat
Agree

○
Strongly
Agree

Bad intervals

How much would I have to pay you to take your phone away for 24 hours?

$20

Ratio scales should have a range that captures the full distribution of responses.

How much would I have to pay you to take your phone away for 24 hours?

$1000

Getting the “best” answers (most accurate and reliable)

Three key questions:

Do we care what they have to say?
Do they know the answer?
Are they willing to tell us the answer?

Each suggests a different reason for why a respondent might be unable or unwilling to answer a survey question.

Inability/unwillingness to answer

Is the respondent qualified?

Qualifications

Screening questions (filter questions) prevent unqualified respondents from being included in the study.

Sometimes combined with a “skip question”, used if the next question (or set of questions) should be responded to only by respondents who satisfy a previous condition.

Are you 21 years of age or older?

○ No → (Interviewer: Ask if someone 21+ is available, or end call)

○ Yes → (Interviewer: Continue)

Inability/unwillingness to answer

Memory aids

Answering questions requires effort. Recall is effort. Writing is effort.

Please list all the departments from which you purchased merchandise on your most recent shopping trip to a department store.

In the list that follows, please check all the departments from which you purchased merchandise on your most recent shopping trip to a department store.

1. Women's dresses☐

2. Men's apparel☐

3. Children's apparel☐

4. Cosmetics☐

...

16. Jewelry☐

17. Other (please specify)☐

Reduced cognitive load

Can the respondent remember?

How many gallons of soft drinks did you consume during the last four weeks?

How often do you consume soft drinks in a typical week?

○
Less than
once a week

○
1 to 3 times
per week

○
4 to 6 times
per week

○
7 or more
times per week

Note: you can also measure soft drinks in bottles/cans.

Do they remember?

Unstructured vs. structured questions 1

What hobbies or activities do you engage in at home?

Structured (multiple choice):

Which of the following activities do you consider to be your hobbies?

☐ Gardening

☐ Home repairs

☐ Athletics

☐ Painting

☐ Reading

☐ Other (Specify) ________

Unstructured vs. structured questions 2

What hobbies or activities do you engage in at home?

Structured (dichotomous):

Do you engage in gardening at home?

○
Yes

○
No

Unstructured vs. structured summary

	Unstructured (Open Ended)	Structured (Closed-Ended)
Examples	• Sentences • Unaided recall	• Multiple choice • Dichotomous • Scaled response
Pros	• Rich array of information • Good for exploratory research when we are unsure what specific questions to ask	• Easier for respondent to answer • Easier to code, tabulate, and interpret data • Less interviewer error
Cons	• Burdensome for respondent • Not good for self-administered surveys • Difficult to analyze	• Less flexible than open-ended (not good for exploratory research)

Inability/unwillingness to answer

Sensitive questions

Respondents may be unwilling to disclose, or unwilling to accurately disclose, sensitive information because this may cause embarrassment or threaten the respondent’s prestige or self-image.

"Binge Drinking" has been defined as consuming five or more alcoholic drinks in one sitting. Have you ever been a binge drinker?

○
Yes

○
No

Have you ever consumed five or more drinks in one sitting? For the purposes of this study, a drink is defined as a bottle of beer, a glass of wine, a wine cooler, a shot glass of liquor, a mixed drink, or a similar drink containing alcohol.

○
Yes

○
No

Projective techniques: third-person

Thinking about last Friday, how many alcoholic drinks did you consume?

○ None

○ 1 drink

○ 2-3 drinks

○ 4-6 drinks

○ More than 6 drinks

On a typical Friday night, approximately how many alcoholic drinks do you think your college classmates usually consume?

○ None

○ 1 drink

○ 2-3 drinks

○ 4-6 drinks

○ More than 6 drinks

Hiding the true nature of the survey

In a study about gambling and drinking behavior in casinos, the researchers want to identify the participants’ gambling habits.

Have you gambled in the past two months?

○
Yes

○
No

Only 1% answered "Yes"

Which of the following things have you done in the past two months?

☐ Gone for a bike ride

☐ Gone to a movie

☐ Gone out for dinner

☐ Gambled — 7% answered "Yes"

☐ Gone to the beach

Will they tell you?

Overcoming unwillingness to answer

Preface the question with a statement that the behavior of interest is common.
Explain why the data are needed.
Ask the question using a projective technique.
Provide response categories rather than asking for specific figures (e.g., income group instead of income in $).
Place sensitive topics at the end of the questionnaire.
Avoid “complex questions” or “double negative questions”.

It depends on how you ask

How do Americans feel about kneeling during the national anthem as an act of protest?

Answer: It depends on how you ask.

Source: fivethirtyeight

What’s in a name?

Obamacare vs. Affordable Care Act

Source: cnbc

Try some out…

In-App Surveys
Email Surveys
- “Survey”
- “Feedback”
- “Experience survey”
- “Customer experience”
- “SMG”, “Clarion”, …
Receipts

Questionnaire structure

Motivating example

Consider these two questions:

Question A: “In selecting a department store, how important is convenience of location?”
Question B: “What considerations are important to you in selecting a department store?”

Should A or B come earlier in the questionnaire? Why?

Questionnaire layout summary

Introduction
Screening questions
- To identify qualified respondents
Research question section
- General to specific (funnel approach)
- Create sections of similar content
- Some sensitive questions can be placed at the end of this section
Last section
- Sensitive topics (e.g., political and religious beliefs, involvement in accidents or crimes)
- Demographics (e.g., money, family life): limit order effects, lower attrition at the beginning

Introduction

"Introduce" the topic that is being studied
- This does not have to be a detailed description (and may even be deliberately vague), but it should be clear enough to set the stage for the questions that follow
Outline expectations
- Purpose
- Expected time
"Legal" language/consent
Mention any incentives (reward for finishing)

Example: Questionnaire flow

Location	Type	Examples	Rationale
Screeners	Screening questions	Do you own a bicycle?	To identify target respondents.
First few	Warm-ups	What brand bicycle do you own?	To ensure the survey is simple and easy to answer. Establish a rapport.
First third	Transitions	What features do you like best about your bicycle?	General questions.
Middle	Difficult and complicated questions	Please rate your bicycle on each of the following dimensions.	The respondent has committed to completing the questionnaire.
Last few	Demographic questions	What is the highest level of education you have attained?	Avoid making respondent feel uneasy early.

Order effects

An order effect is a systematic change in survey responses caused by a question’s position within the survey.

Why it happens:

Learning: answering one question can change how respondents interpret and answer subsequent questions
- Context/priming: earlier questions change what's "top of mind"
- Anchoring: early numbers/ideas become a reference point
Fatigue: later questions get less attention
- Place most important questions in the middle

Incentives

Incentives are offers to do something for the respondent to increase the probability that the respondent will participate in the survey.
Can be monetary or non-monetary
Non-monetary incentives:
- Anonymity: respondent name not identified
- Confidentiality: respondent's name is known by the researcher but not divulged to a third party
- SHORT!!: "The survey will take less than five minutes to complete."

Test. Test. Test. Finalize.

Start with a dry run of the survey on a small, representative set of respondents
Can also use "bots" to generate test responses, but you ALWAYS want human feedback as well
- On Qualtrics, go to Tools → Generate test responses
Goal: Reveal errors before the survey is launched
- Ask for feedback on wording, clarity
- Does the data look as expected?
The longer and more complex the questionnaire, the greater the need for extensive pretesting

Checklists

Conventional wisdom for survey questions (Krosnick)

Best practices for question wording:

Use simple, familiar words (avoid technical terms, jargon, and slang)
Use simple syntax
Avoid words with ambiguous meanings, i.e., aim for wording that all respondents will interpret in the same way
Strive for wording that is specific and concrete (as opposed to general and abstract)
Make response options exhaustive and mutually exclusive
Avoid leading or loaded questions that push respondents toward an answer
Ask about one thing at a time (avoid double-barreled questions)
Avoid questions with single or double negations

Conventional wisdom for survey structure (Krosnick)

Best practices for survey structure:

Early questions should be easy and pleasant to answer, and should build rapport between the respondent and the researcher
Questions at the very beginning of a questionnaire should explicitly address the topic of the survey, as it was described to the respondent prior to the interview
Questions on the same topic should be grouped together
Questions on the same topic should proceed from general to specific
Questions on sensitive topics that might make respondents uncomfortable should be placed at the end of the questionnaire
Filter questions should be included, to avoid asking respondents questions that do not apply to them

Examples from my inbox

Let’s look at how professional organizations structure their surveys:

HBO Max customer satisfaction survey
Ad Age subscription cancellation survey

Max

Max: Introduction (& incentives)

Max: Relevance check

Max: Easy general questions

Max: More detailed questions

Max: Consumer characteristics

Max: Consumer preferences

Max: Demographics

Max: Thank you and submit

Ad Age: Email introduction

Ad Age: Cancellation reasons

Ad Age: Likelihood to resubscribe

Ad Age: Resubscribe timeline

Ad Age: Moving to specific Qs

Ad Age: More detailed questions

Ad Age: Back to easy

Ad Age: Demographics

Ad Age: Thank you

More housekeeping

Thinking Forward

Feb 20, 12:30-5pm
In-person in this building
Email went out with additional info and RSVP link
Small commitment assignment on D2L
- Review the list of speakers
- Choose sessions to attend
- Come up with a question to ask one of the speakers

GA1

Excellent work!
- Every team submitted an idea that has the potential to be a great project
- My comments are intended to steer you toward a more focused research question and a more feasible project scope
- I am excited to see how these ideas evolve over the rest of the semester!
Detailed feedback
- I read and have uploaded comments on your submissions
- See attached PDFs on assignment page
- My thoughts on feedback

Submission guidelines for future assignments

Don’t include the prompts (question numbers, instructions, etc.) in your submission
- Combine your “answers” into a cohesive narrative
- Or, label sections with descriptive headings (e.g., “Research question and motivation”, “Driving challenge”, “Consumer insights”, etc.)
Structure/formatting is entirely up to you
Where possible, combine your submission into a single document (PDF)

Rest of today

Hand back last week’s assignment
New in-class assignment to revise your questions from last week
I will check in with each team
- Questions on GA1 feedback (my handwriting, direction, etc.)?
- How is GA2 going?